Incremental CFG Parsing with Statistical Lexical Dependencies

Incremental parsing with a context free grammar produces partial syntactic structures for an initial fragment on the word-by-word basis. Owing to the syntactic ambiguity, however, too many structures are produced, and therefore its parsing speed becomes very slow. This paper describes a technique for efficient incremental parsing using lexical information. The probability concerning dependencies between words, as the lexical information, is automatically acquired from a large-scale corpus with syntactic structures. A process for discarding syntactic structures which will not be likely has been integrated into the incremental chart parsing. That is, partial syntactic structures whose dependency probabilities are not high will be removed from the chart. Our technique proposed in this paper can also be considered as a kind of practical methods of incremental disambiguation. An experiment using Penn Treebank has shown our technique to be feasible and efficient.